Improving Scene Graph Classification by Exploiting Knowledge from Texts

Training scene graph classification models requires a large amount of annotated image data. Meanwhile, graphs represent relational knowledge that can be modeled with symbolic data from texts or graphs. While annotation demands extensive labor, collecting textual descriptions natural scenes less effort. In this work, we investigate whether substitute for To end, employ framework is trained not only images but also our architecture, the entities are first mapped to their correspondent image-grounded representations and then fed into reasoning pipeline. Even though structured form knowledge, such as in graphs, always available, generate it unstructured using transformer-based language model. We show by fine-tuning pipeline extracted texts, achieve ~8x more accurate results classification, ~3x object ~1.5x predicate compared supervised baselines 1% images.